{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8927f995-24ad-4c1e-81b1-47daec56023b",
   "metadata": {},
   "source": [
    "# 词的相似性和类比任务\n",
    ":label:`sec_synonyms`\n",
    "\n",
    "在 :numref:`sec_word2vec_pretraining`中，我们在一个小的数据集上训练了一个word2vec模型，并使用它为一个输入词寻找语义相似的词。实际上，在大型语料库上预先训练的词向量可以应用于下游的自然语言处理任务，这将在后面的 :numref:`chap_nlp_app`中讨论。为了直观地演示大型语料库中预训练词向量的语义，让我们将预训练词向量应用到词的相似性和类比任务中。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b30a565a-945f-4122-b408-77a1a1109270",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('..')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ad25d092-ecb2-42a0-9e48-6716620c6ff6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import mindspore\n",
    "import mindspore.nn as nn\n",
    "import mindspore.ops as ops\n",
    "from d2l import mindspore as d2l"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c618f34-ef01-4c7b-b961-5c5c3745c834",
   "metadata": {},
   "source": [
    "## 加载预训练词向量\n",
    "\n",
    "以下列出维度为50、100和300的预训练GloVe嵌入，可从[GloVe网站](https://nlp.stanford.edu/projects/glove/)下载。预训练的fastText嵌入有多种语言。这里我们使用可以从[fastText网站](https://fasttext.cc/)下载300维度的英文版本（“wiki.en”）。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "391532d5-0267-4018-bb83-77db48c58ea9",
   "metadata": {},
   "outputs": [],
   "source": [
    "#@save\n",
    "d2l.DATA_HUB['glove.6b.50d'] = (d2l.DATA_URL + 'glove.6B.50d.zip',\n",
    "                                '0b8703943ccdb6eb788e6f091b8946e82231bc4d')\n",
    "\n",
    "#@save\n",
    "d2l.DATA_HUB['glove.6b.100d'] = (d2l.DATA_URL + 'glove.6B.100d.zip',\n",
    "                                 'cd43bfb07e44e6f27cbcc7bc9ae3d80284fdaf5a')\n",
    "\n",
    "#@save\n",
    "d2l.DATA_HUB['glove.42b.300d'] = (d2l.DATA_URL + 'glove.42B.300d.zip',\n",
    "                                  'b5116e234e9eb9076672cfeabf5469f3eec904fa')\n",
    "\n",
    "#@save\n",
    "d2l.DATA_HUB['wiki.en'] = (d2l.DATA_URL + 'wiki.en.zip',\n",
    "                           'c1816da3821ae9f43899be655002f6c723e91b88')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e35f192-a2d5-4cdf-acf2-077107fadead",
   "metadata": {},
   "source": [
    "为了加载这些预训练的GloVe和fastText嵌入，我们定义了以下`TokenEmbedding`类。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3884891a-958e-41b3-8135-a4a47253258f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#@save\n",
    "class TokenEmbedding:\n",
    "    \"\"\"GloVe嵌入\"\"\"\n",
    "    def __init__(self, embedding_name):\n",
    "        self.idx_to_token, self.idx_to_vec = self._load_embedding(\n",
    "            embedding_name)\n",
    "        self.unknown_idx = 0\n",
    "        self.token_to_idx = {token: idx for idx, token in\n",
    "                             enumerate(self.idx_to_token)}\n",
    "\n",
    "    def _load_embedding(self, embedding_name):\n",
    "        idx_to_token, idx_to_vec = ['<unk>'], []\n",
    "        data_dir = d2l.download_extract(embedding_name)\n",
    "        # GloVe网站：https://nlp.stanford.edu/projects/glove/\n",
    "        # fastText网站：https://fasttext.cc/\n",
    "        with open(os.path.join(data_dir, 'vec.txt'), 'r') as f:\n",
    "            for line in f:\n",
    "                elems = line.rstrip().split(' ')\n",
    "                token, elems = elems[0], [float(elem) for elem in elems[1:]]\n",
    "                # 跳过标题信息，例如fastText中的首行\n",
    "                if len(elems) > 1:\n",
    "                    idx_to_token.append(token)\n",
    "                    idx_to_vec.append(elems)\n",
    "        idx_to_vec = [[0] * len(idx_to_vec[0])] + idx_to_vec\n",
    "        return idx_to_token, mindspore.Tensor(idx_to_vec)\n",
    "\n",
    "    def __getitem__(self, tokens):\n",
    "        indices = [self.token_to_idx.get(token, self.unknown_idx)\n",
    "                   for token in tokens]\n",
    "        vecs = self.idx_to_vec[mindspore.Tensor(indices)]\n",
    "        return vecs\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.idx_to_token)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5083c0e-2ebc-48ae-bc2e-40caccf5e940",
   "metadata": {},
   "source": [
    "下面我们加载50维GloVe嵌入（在维基百科的子集上预训练）。创建`TokenEmbedding`实例时，如果尚未下载指定的嵌入文件，则必须下载该文件。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b564218f-e235-4639-9f50-e9371dd990fe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在从http://d2l-data.s3-accelerate.amazonaws.com/glove.6B.50d.zip下载../data/glove.6B.50d.zip...\n"
     ]
    }
   ],
   "source": [
    "glove_6b50d = TokenEmbedding('glove.6b.50d')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa3776a0-c0bf-422c-b97e-0eeadcde04b8",
   "metadata": {},
   "source": [
    "输出词表大小。词表包含400000个词（词元）和一个特殊的未知词元。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f2efd3a3-e101-45b2-ad97-856fcc4f31de",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "400001"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(glove_6b50d)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a33f3cc8-29c7-4f2b-be6c-55af48792efa",
   "metadata": {},
   "source": [
    "我们可以得到词表中一个单词的索引，反之亦然。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6bdee7a7-9030-4ec0-bf94-efe8340196f0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3367, 'beautiful')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "glove_6b50d.token_to_idx['beautiful'], glove_6b50d.idx_to_token[3367]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c788e88-27dc-4458-82bf-6a38b7917c70",
   "metadata": {},
   "source": [
    "## 应用预训练词向量\n",
    "\n",
    "使用加载的GloVe向量，我们将通过下面的词相似性和类比任务中来展示词向量的语义。\n",
    "\n",
    "### 词相似度\n",
    "\n",
    "与 :numref:`subsec_apply-word-embed`类似，为了根据词向量之间的余弦相似性为输入词查找语义相似的词，我们实现了以下`knn`（$k$近邻）函数。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9f78e8a8-2a5a-4955-995b-319775735bc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def knn(W, x, k):\n",
    "    # 增加1e-9以获得数值稳定性\n",
    "    cos = ops.mv(W, x.reshape(-1,)) / (\n",
    "        ops.sqrt(ops.sum(W * W, dim=1) + 1e-9) *\n",
    "        ops.sqrt((x * x).sum()))\n",
    "    _, topk = ops.topk(cos, k=k)\n",
    "    return topk, [cos[int(i)] for i in topk]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90de12b5-d88c-456b-8782-b67cf761904f",
   "metadata": {},
   "source": [
    "然后，我们使用`TokenEmbedding`的实例`embed`中预训练好的词向量来搜索相似的词。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a35a5991-6692-46be-93de-a44c794093ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_similar_tokens(query_token, k, embed):\n",
    "    topk, cos = knn(embed.idx_to_vec, embed[[query_token]], k + 1)\n",
    "    for i, c in zip(topk[1:], cos[1:]):  # 排除输入词\n",
    "        print(f'{embed.idx_to_token[int(i)]}：cosine相似度={float(c):.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d91b3962-8461-4937-9f8c-4dc702ea13fb",
   "metadata": {},
   "source": [
    "`glove_6b50d`中预训练词向量的词表包含400000个词和一个特殊的未知词元。排除输入词和未知词元后，我们在词表中找到与“chip”一词语义最相似的三个词。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6345ef84-935f-4a88-a795-3bc72a6aa789",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chips：cosine相似度=0.856\n",
      "intel：cosine相似度=0.749\n",
      "electronics：cosine相似度=0.749\n"
     ]
    }
   ],
   "source": [
    "get_similar_tokens('chip', 3, glove_6b50d)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33dbceaf-c7fa-4f31-a47a-8299d5f9547c",
   "metadata": {},
   "source": [
    "下面输出与“baby”和“beautiful”相似的词。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8116bf17-15b9-4c42-a7fe-6fe8513d7cd0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "babies：cosine相似度=0.839\n",
      "boy：cosine相似度=0.800\n",
      "girl：cosine相似度=0.792\n"
     ]
    }
   ],
   "source": [
    "get_similar_tokens('baby', 3, glove_6b50d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "de0aa4c4-ed83-49f3-ad02-9012ef8c468f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lovely：cosine相似度=0.921\n",
      "gorgeous：cosine相似度=0.893\n",
      "wonderful：cosine相似度=0.830\n"
     ]
    }
   ],
   "source": [
    "get_similar_tokens('beautiful', 3, glove_6b50d)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a269402-79af-49c4-950b-91a9cbd583d6",
   "metadata": {},
   "source": [
    "### 词类比\n",
    "\n",
    "除了找到相似的词，我们还可以将词向量应用到词类比任务中。\n",
    "例如，“man” : “woman” :: “son” : “daughter”是一个词的类比。\n",
    "“man”是对“woman”的类比，“son”是对“daughter”的类比。\n",
    "具体来说，词类比任务可以定义为：\n",
    "对于单词类比$a : b :: c : d$，给出前三个词$a$、$b$和$c$，找到$d$。\n",
    "用$\\text{vec}(w)$表示词$w$的向量，\n",
    "为了完成这个类比，我们将找到一个词，\n",
    "其向量与$\\text{vec}(c)+\\text{vec}(b)-\\text{vec}(a)$的结果最相似。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b6878109-e0c8-4055-8b30-62415e48a6f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_analogy(token_a, token_b, token_c, embed):\n",
    "    vecs = embed[[token_a, token_b, token_c]]\n",
    "    x = vecs[1] - vecs[0] + vecs[2]\n",
    "    topk, cos = knn(embed.idx_to_vec, x, 1)\n",
    "    return embed.idx_to_token[int(topk[0])]  # 删除未知词"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92077949-79ac-4bd7-91cf-f1c6f364b1c8",
   "metadata": {},
   "source": [
    "让我们使用加载的词向量来验证“male-female”类比。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5921dea4-1fa0-479c-9be1-6e9482634bb6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'daughter'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_analogy('man', 'woman', 'son', glove_6b50d)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5109ee16-95a9-460a-9759-b70d528743b4",
   "metadata": {},
   "source": [
    "下面完成一个“首都-国家”的类比：\n",
    "“beijing” : “china” :: “tokyo” : “japan”。\n",
    "这说明了预训练词向量中的语义。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "50d653f3-a056-4cb8-afed-e87b4c35fb32",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'japan'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_analogy('beijing', 'china', 'tokyo', glove_6b50d)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c32389b-4366-423b-ab26-f3ba3562eb12",
   "metadata": {},
   "source": [
    "另外，对于“bad” : “worst” :: “big” : “biggest”等“形容词-形容词最高级”的比喻，预训练词向量可以捕捉到句法信息。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ae5c8e02-b98d-4fff-b359-ebe4bd519de6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'biggest'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_analogy('bad', 'worst', 'big', glove_6b50d)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c92eee53-49e2-465e-8d92-c95296a7af85",
   "metadata": {},
   "source": [
    "为了演示在预训练词向量中捕捉到的过去式概念，我们可以使用“现在式-过去式”的类比来测试句法：“do” : “did” :: “go” : “went”。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "2915eaf7-f393-4137-870d-bbd26835744d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'went'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_analogy('do', 'did', 'go', glove_6b50d)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2e86de6-1b67-4f94-90d3-678687f20731",
   "metadata": {},
   "source": [
    "## 小结\n",
    "\n",
    "* 在实践中，在大型语料库上预先练的词向量可以应用于下游的自然语言处理任务。\n",
    "* 预训练的词向量可以应用于词的相似性和类比任务。\n",
    "\n",
    "## 练习\n",
    "\n",
    "1. 使用`TokenEmbedding('wiki.en')`测试fastText结果。\n",
    "1. 当词表非常大时，我们怎样才能更快地找到相似的词或完成一个词的类比呢？\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "593c1ef2-3c47-410b-80dc-3a488abda31b",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ms_d2l",
   "language": "python",
   "name": "ms_d2l"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
